.. code:: ipython3 from seeq import spy import pandas as pd # Set the compatibility option so that you maximize the chance that SPy will remain compatible with your notebook/script spy.options.compatibility = 192 .. code:: ipython3 # Log into Seeq Server if you're not using Seeq Data Lab: spy.login(url='http://localhost:34216', credentials_file='../credentials.key', force=False) Asset Trees 2: Templates ======================== In the :doc:`first asset trees tutorial notebook `, we learned how to use the ``spy.assets.Tree`` Python class to define, modify, and push asset trees to Seeq. In this notebook, we will dive deeper into the ``spy.assets`` submodule and explore its ability to create asset trees out of customized Python “templates”. You may have heard of the concept of a `Digital Twin `__, which is a “virtual representation that serves as the real-time digital counterpart of a physical object or process.” The ``spy.assets`` submodule provides a framework for defining Digital Twins using the power and relative ease-of-use of the Python programming language. You can leverage object-oriented design principles like *encapsulation*, *inheritance*, *composition*, and *mixins* to increase reuse and consistency while still accommodating the many exceptions that naturally occur in manufacturing scenarios. As with other capabilities in SPy, the work you do with ``spy.assets`` is (by default) sandboxed to a particular workbook that you own. As a result, you can easily experiment with new asset structures and calculations before publishing to your broader organization. Simple Trees vs Templates ------------------------- When should you use the ``spy.assets.Tree()`` and when should you use the Template methods described here? Start with the simpler ``spy.assets.Tree()`` functionality unless, as you skim through this documentation, it’s clear that you’d like to try the Template functionality first. You can always “graduate” to this more advanced approach. Concepts -------- There are several important concepts to understand as we dive in to the ``spy.assets`` submodule: - An **asset** is simply a container (with associated *properties*) that has children. Those children can be other assets and/or *attributes* (signals, conditions and scalars). - An **attribute** is a signal, condition or scalar that is a child of an asset and is said to be *contained* by that asset. This is a similar concept to OSIsoft Asset Framework’s *attribute*. - A **property** is a named value (with optional units of measure) that captures metadata associated with assets, signals, conditions and scalars. They correspond to the columns in a metadata DataFrame. - A **component** is an asset that is a child of another asset and is said to be *contained* by the parent asset. For example, a *Furnace* asset may contain a *Blast Air Blower*. - A **template** defines the set of *attributes* and *components* that comprise a particular Asset. A template corresponds to a class of asset like *Furnace*, *Well*, *Pump*. - A **reference** is a link to a signal, condition or scalar that exists somewhere else. If you are mapping a flat list of tags into an asset hierarchy, the asset hierarchy contains *references* to those tags. How Python fits in ------------------ A template is specified in Python by defining a `class `__ that derives from ``spy.assets.Asset`` or ``spy.assets.Mixin`` (more about *mixins* later). These classes can have member functions and data members, just like any other Python class. Classes can have special member functions that represent *attributes*. They are *decorated* with the ``spy.assets.Asset.Attribute`` decorator, which tells the SPy framework to use them to define attributes for the asset. Similarly, the ``spy.assets.Asset.Component`` decorated indicates to the SPy framework that a *component* is being defined as a child of the asset, which is usually a different Python class instance that derives from ``spy.assets.Asset`` or ``spy.assets.Mixin``. You’ll see how it works in the examples that follow. Getting Started --------------- First we need to import the Python modules we will need and log in: .. code:: ipython3 from seeq.spy.assets import Asset, ItemGroup # Show all data in DataFrame output -- don't truncate it pd.set_option('display.max_colwidth', None) Preparing the Ingredients ------------------------- There are two main steps to creating an asset tree, and they’re similar to cooking in your kitchen: First you find and prepare the “ingredients”, then you use them in a “recipe”. The “ingredients” are the signals, conditions and/or scalars that already exist in Seeq, either as indexed items from external datasources or as items you’ve imported to the internal Seeq database. You will create a DataFrame that represents the *metadata* ingredients, and those ingredients will be used by ``spy.assets.build()`` to create another DataFrame full of new items to be pushed as an asset tree. For this example, we’re going to map a flat tag structure into an asset template called ``HVAC``. We will use Seeq’s built-in Example Data. Let’s search for all the “flat list” example data tags that have the pattern ``Area _``: .. code:: ipython3 hvac_metadata_df = spy.search({ 'Name': 'Area ?_*' }) hvac_metadata_df.head() The ``hvac_metadata_df`` *metadata* DataFrame serves as the “ingredients” for the recipe that we will define a little later. In order to build the HVAC assets, we must add three important columns to our *metadata* DataFrame: - ``Build Asset`` specifies the name of the asset that a row of metadata applies to. In our case, that will be ``Area X``, where X is a letter differentiating the different areas of the plant that an HVAC system serves. - ``Build Path`` specifies the path through the asset hierarchy where you want the asset to live. We will create these columns using the power of Pandas DataFrames: .. code:: ipython3 # We can use Pandas' string extraction capabilities to create the Build Asset column hvac_metadata_df['Build Asset'] = hvac_metadata_df['Name'].str.extract('(Area .)_.*') # We will specify a simple path in a new tree where we want these to live hvac_metadata_df['Build Path'] = 'My HVAC Units >> Facility #1' hvac_metadata_df Writing the Recipe ------------------ The recipe that turns the ingredients into an asset structure is specified using Python classes that derive from ``Asset`` or ``Mixin``. First let’s define our ``HVAC`` class: .. code:: ipython3 class HVAC(Asset): @Asset.Attribute() def Temperature(self, metadata): # We use simple Pandas syntax to select for a row in the DataFrame corresponding to our desired tag return metadata[metadata['Name'].str.endswith('Temperature')] @Asset.Attribute() def Relative_Humidity(self, metadata): # All Attribute functions must take (self, metadata) as parameters return metadata[metadata['Name'].str.contains('Humidity')] This Python code defines an ``HVAC`` asset that has two attributes: ``Temperature`` and ``Relative Humidity``. These attributes are represented in Python as functions that can return any of the following: 1. A single-row DataFrame containing an ID column that identifies an item (signal/condition/scalar/metric) to expose on this asset. (As seen above.) 2. A dictionary that defines the item. A ``Type`` entry is required (whose value must be ``Signal``, ``Condition``, ``Scalar`` or ``Metric``). If ``Name`` is not supplied, the function name will be used as the ``Name`` with any underscores automatically replaced with a space. As you’ll see below, you can specify ``Formula`` and ``Formula Parameters`` if you are trying to specify a calculated item. 3. A *list* of dictionaries if you want to specify multiple items in the format of (2) above. Now we can feed the “ingredients” into the “recipe” using the ``spy.assets.build()`` function. This command will build a new set of asset and signal definitions based on the ``hvac_metadata_df`` *metadata* DataFrame. Each unique combination of ``Build Path`` and ``Build Asset`` will be treated as a different asset, and the ``metadata`` argument that is passed in to the ``Asset.Attribute()`` decorated functions will only contain the rows for a particular ``Build Path`` / ``Build Asset`` pair. .. code:: ipython3 build_df = spy.assets.build(HVAC, hvac_metadata_df) build_df # NOTE: # # There will be errors in this example, the status table will be colored red. Read more below. In the progress table above (which should be colored red), if you look at the ``Build Result`` column for Area F, you can see that no matching metadata was found. If you then do a search in Seeq Workbench for ``Area F_``, you’ll see that there are no ``Temperature`` or ``Relative Humidity`` tags for that area. That’s fine! When we push, we simply won’t add signals for Area F. The new ``build_df`` DataFrame contains new metadata that represent all the Seeq items that can be pushed into Seeq to realize this simple asset model. .. code:: ipython3 spy.push(metadata=build_df, workbook='SPy Documentation Examples >> spy.assets') The asset tree is now available for viewing in Seeq. It is scoped to a workbook that you own called *SPy Documentation Examples >> spy.assets*, so you won’t see it in any other workbook. If/when you want to publish the tree globally, add the ``workbook=None`` parameter to your ``spy.push()`` function call. Calculated Signals, Conditions and Scalars ------------------------------------------ Now let’s do something a little more advanced. We’ll define a new class that has calculations alongside references to raw signals. This class will *derive from* our existing ``HVAC`` class so that it *inherits* the references we already defined. .. code:: ipython3 class HVAC_With_Calcs(HVAC): @Asset.Attribute() def Temperature_Rate_Of_Change(self, metadata): return { 'Type': 'Signal', # This formula will give us a nice derivative in F/h 'Formula': '$temp.lowPassFilter(150min, 3min, 333).derivative() * 3600 s/h', 'Formula Parameters': { # We can reference the base class' Temperature attribute here as a dependency '$temp': self.Temperature(), } } @Asset.Attribute() def Too_Hot(self, metadata): return { 'Type': 'Condition', 'Formula': '$temp.valueSearch(isGreaterThan($threshold))', 'Formula Parameters': { '$temp': self.Temperature(), # We can also reference other attributes in this derived class '$threshold': self.Hot_Threshold() } } @Asset.Attribute() def Hot_Threshold(self, metadata): return { 'Type': 'Scalar', 'Formula': '80F' } We’ll make some adjustments to our metadata DataFrame to use the new template and then push into Seeq: .. code:: ipython3 # Make a copy of our original metadata but change the Build Template hvac_with_calcs_metadata_df = hvac_metadata_df.copy() build_with_calcs_df = spy.assets.build(HVAC_With_Calcs, hvac_with_calcs_metadata_df) pd.set_option('display.max_colwidth', 50) spy.push(metadata=build_with_calcs_df, workbook='SPy Documentation Examples >> spy.assets') Now when you look at the asset tree in Seeq, you’ll see ``Temperature Rate Of Change``, ``Too Hot`` and ``Hot Threshold``. Each time we push, we are overwriting the definition we pushed previously. Components ---------- You can create an entire asset hierarchy by defining Asset classes that are *composed* of other Asset classes. To illustrate, let’s start by preparing a DataFrame with our ingredients. This example will be a little contrived given the example data that is available in Seeq. We will define a metadata DataFrame where there are two Refrigerators and each will have two Compressors. In the cell below, you will see that we define the ``Build Asset`` column with the Refigerator name but also define a ``Compressor`` column with the Compressor name. We have assigned Area A-D signals to specific Refrigerators and Compressors. .. code:: ipython3 metadata_df = spy.search({ 'Name': r'/Area [A-D]_(?:Temperature|Compressor Power)/', 'Datasource Class': 'Time Series CSV Files' }) metadata_df.loc[metadata_df['Name'] == 'Area A_Temperature', 'Build Asset'] = 'Refrigerator 1' metadata_df.loc[metadata_df['Name'] == 'Area A_Compressor Power', 'Build Asset'] = 'Refrigerator 1' metadata_df.loc[metadata_df['Name'] == 'Area A_Compressor Power', 'Compressor'] = 'Compressor 1' metadata_df.loc[metadata_df['Name'] == 'Area B_Compressor Power', 'Build Asset'] = 'Refrigerator 1' metadata_df.loc[metadata_df['Name'] == 'Area B_Compressor Power', 'Compressor'] = 'Compressor 2' metadata_df.loc[metadata_df['Name'] == 'Area C_Temperature', 'Build Asset'] = 'Refrigerator 2' metadata_df.loc[metadata_df['Name'] == 'Area C_Compressor Power', 'Build Asset'] = 'Refrigerator 2' metadata_df.loc[metadata_df['Name'] == 'Area C_Compressor Power', 'Compressor'] = 'Compressor 3' metadata_df.loc[metadata_df['Name'] == 'Area D_Compressor Power', 'Build Asset'] = 'Refrigerator 2' metadata_df.loc[metadata_df['Name'] == 'Area D_Compressor Power', 'Compressor'] = 'Compressor 4' metadata_df['Build Path'] = 'Refrigerator Units' metadata_df[['ID', 'Name', 'Build Path', 'Build Asset', 'Compressor']] Now we can define our Asset classes. Will be using the ``Asset.Component()`` decorator and the ``self.build_components()`` function: .. code:: ipython3 class Refrigerator(Asset): @Asset.Attribute() def Temperature(self, metadata): # This signal attribute is assigned to the Refrigerator asset return metadata[metadata['Name'].str.endswith('Temperature')] # Note the use of Asset.Component here, which allows us to return a list of definitions # instead of just a single definition. @Asset.Component() def Compressors(self, metadata): # Using the Compressor template class, we build all of the compressor definitions # associated with a particular Refrigerator. The column_name supplied tells the # build_components function which metadata column to use for the Compressor names. return self.build_components(template=Compressor, metadata=metadata, column_name='Compressor') @Asset.Attribute() def Compressor_Power_Max(self, metadata): # We can refer to the Compressors and "pick" attributes for which to perform a # roll up. In this example, we're picking the 'Power' signals that are on each # compressor and creating a new signal representing the maximum power across # all the compressors. return self.Compressors().pick({ 'Name': 'Power' }).roll_up('maximum') @Asset.Attribute() def Compressor_High_Power(self, metadata): # Similar to Compressor_Power_Max, we are rolling up a compressor calculation but # this time it's a condition. 'High Power' at the Refrigerator level will have # capsules if either compressor's 'High Power' condition is present. # # This time we'll use a different method of picking the child items than we used # in Compressor_Power_Max() above. In this case, we're going to select the set of # compressors that are owned by this asset from the entire set of assets, and use # Python conditional logic to find the "High Power" conditions. What you see # below is called a Python "list comprehension" that combines iteration over all # assets (the "for/in" construct) with filtering (the "if" statement). # # Helpful functions: # asset.is_child_of(self) - Is the asset one of my direct children? # asset.is_parent_of(self) - Is the asset my direct parent? # asset.is_descendant_of(self) - Is the asset below me in the tree? # asset.is_ancestor_of(self) - Is the asset above me? (i.e. parent/grandparent/great-grandparent/etc) # return ItemGroup([ asset.High_Power() for asset in self.all_assets() if asset.is_child_of(self) ]).roll_up('union') class Compressor(Asset): @Asset.Attribute() def Power(self, metadata): # Each compressor has just a single attribute, Power return metadata[metadata['Name'].str.endswith('Power')] @Asset.Attribute() def High_Power(self, metadata): return { 'Type': 'Condition', 'Formula': '$a.valueSearch(isGreaterThan(20kW))', 'Formula Parameters': { '$a': self.Power() } } @Asset.Attribute() def Other_Compressors_Are_High_Power(self, metadata): # This is a more complex example of using self.all_assets() where we want to # look at sibling assets as opposed to parents/children, and do a roll up. # Here the "if" statement selects Compressor assets where our parent and # their parent are the same but we exclude ourselves. return ItemGroup([ asset.High_Power() for asset in self.all_assets() if isinstance(asset, Compressor) and self.parent == asset.parent and self != asset ]).roll_up('union') build_df = spy.assets.build(Refrigerator, metadata_df, errors='raise') spy.push(metadata=build_df, workbook='SPy Documentation Examples >> spy.assets') There should now be a ``Refrigerator Units`` asset tree in Seeq with two Refrigerators with two Compressors each. In this manner, you can make an arbitrarily deep hierarchy composed of various asset types. Metrics ------- Metrics (aka *Scorecard Metrics* / *Threshold Metrics*) are powerful items in Seeq that allow you to easily specify aggregations, statistics and associated boundaries. Click on ``Scorecard Metric`` in the Seeq Workbench *Tools* pane to experiment with them. Metrics can be specified as attributes in asset classes and can refer to other attributes. Here are some examples: .. code:: ipython3 class HVAC_With_Metrics(HVAC): @Asset.Attribute() def Too_Humid(self, metadata): return { 'Type': 'Condition', 'Name': 'Too Humid', 'Formula': '$relhumid.valueSearch(isGreaterThan(70%))', 'Formula Parameters': { '$relhumid': self.Relative_Humidity(), } } @Asset.Attribute() def Humidity_Upper_Bound(self, metadata): return { 'Type': 'Signal', 'Name': 'Humidity Upper Bound', 'Formula': '$relhumid + 10', 'Formula Parameters': { '$relhumid': self.Relative_Humidity(), } } @Asset.Attribute() def Humidity_Lower_Bound(self, metadata): return { 'Type': 'Signal', 'Name': 'Humidity Lower Bound', 'Formula': '$relhumid - 10', 'Formula Parameters': { '$relhumid': self.Relative_Humidity(), } } @Asset.Attribute() def Humidity_Statistic_KPI(self, metadata): return { 'Type': 'Metric', 'Measured Item': self.Relative_Humidity(), 'Statistic': 'Range' } @Asset.Attribute() def Humidity_Simple_KPI(self, metadata): return { 'Type': 'Metric', 'Measured Item': self.Relative_Humidity(), 'Thresholds': { 'HiHi': self.Humidity_Upper_Bound(), 'LoLo': self.Humidity_Lower_Bound() } } @Asset.Attribute() def Humidity_Condition_KPI(self, metadata): return { 'Type': 'Metric', 'Measured Item': self.Relative_Humidity(), 'Statistic': 'Maximum', 'Bounding Condition': self.Too_Humid(), 'Bounding Condition Maximum Duration': '30h' } @Asset.Attribute() def Humidity_Continuous_KPI(self, metadata): return { 'Type': 'Metric', 'Measured Item': self.Relative_Humidity(), 'Statistic': 'Minimum', 'Duration': '6h', 'Period': '4h', 'Metric Neutral Color':'#189E4D', #hex color codes can be optionally appended to thresholds 'Thresholds': { 'HiHiHi#FF0000': 60, 'HiHi': 40, 'LoLo#0000ff': 20 } } Now let’s push a small asset tree with these metrics in it: .. code:: ipython3 metadata_df = spy.search({ 'Name': 'Area A_*', 'Datasource Class': 'Time Series CSV Files' }) metadata_df['Build Asset'] = 'Metrics Area A' metadata_df['Build Path'] = 'Metrics Example' build_df = spy.assets.build(HVAC_With_Metrics, metadata_df) push_df = spy.push(metadata=build_df, workbook='SPy Documentation Examples >> spy.assets') You should now see a ``Metrics Example`` tree that you can drill into and bring up metrics. Troubleshooting / Debugging --------------------------- Since we are using Python classes to describe our asset tree and the attributes therein, troubleshooting our code is a little bit harder than just working with DataFrames. If the code within your ``@Asset.Attribute``-decorated function isn’t working the way you expect, ideally you would be able to see what’s happening inside of it while the ``spy.assets.build()`` function is running. A useful tool is Python’s built-in command-line debugger called `pdb `__. Let’s try using it in the example below. When you execute the following cell, you’ll notice that an ``ipdb>`` prompt appears, showing you the line of code you’re about to execute. You can show the value of the ``metadata`` variable just by typing ``metadata`` and pressing ENTER. Then type ``c`` and hit ENTER to allow execution to continue. .. code:: ipython3 # You must import the IPython breakpoint function, called "set_trace" from IPython.core.debugger import set_trace class DebuggingExample(Asset): @Asset.Attribute() def My_Scalar(self, metadata): # Put in a "breakpoint" so that execution stops here and a command line pops up. Notice the # if statement here as an example of how to be more precise about when the breakpoint is hit: # We're only going to enter the debugger if the asset's name is 'Debugging Area A' if self.definition['Name'] == 'Debugging Area A': set_trace() return { 'Type': 'Scalar', 'Formula': '%s' % len(metadata) } debugging_metadata_df = pd.DataFrame([{ 'Build Asset': 'Debugging Area A', 'Build Path': 'Debugging Example' }]) build_df = spy.assets.build(DebuggingExample, debugging_metadata_df) There are lots of great commands to help you navigate through your code as it executes. Read through all the commands at `pdb `__ to familiarize yourself. The most important ones are ``step``, ``next`` and ``continue``. Anything else you type at the prompt gets evaluated as Python code, so you can do things like ``metadata['Build Asset']`` to show just the ``Build Asset`` column of the ``metadata`` DataFrame. Using an Integrated Development Environment (IDE) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you want to “level up” and start using more powerful development and debugging tools, you can! There are several good choices available to you, including `SPyder `__ (free and open source) and `PyCharm `__ (Community Edition is free). If you take this route, you’ll want to move your code into “normal” .py files and execute SPy commands from a main script with the debugger engaged. Detailed Help ------------- All SPy functions have detailed documentation to help you use them. Just execute ``help(spy.)`` like you see below. .. code:: ipython3 .. code:: ipython3 help(spy.assets.build) API Reference Links ------------------- - :py:mod:`seeq.spy.assets`